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Information theory on a time-discrete setting in the framework of time series analysis is generalized 
to the time-continuous case. Considerations of the Roessler and Lorenz dynamics as well as the 
Ornstein-Uhlenbeck process yield for time-continuous entropies a new possibility for the distinction 
of chaos and noise. In the deterministic case an upper threshold of the joint uncertainty in the limit 
of infinitely high sampling rate can be found and the entropy rate can be calculated as a usual time 
derivative of the entropy. In a three-dimensional representation the dependence of the joint entropy 
H (e, r, t) on space resolution, discretization time step length and uncertainty-assessed time is shown 
in a unified manner. Hence the dimension and the Kolmogorov-Sinai entropy rate of any dynamics 
can be read out as limit cases from one single graph. 
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^ ■ I. INTRODUCTION 

£^ ' Uncertainty of the outcome of random variables X is usually evaluated by Shannon entropies [l[ 

H(X) = -Y^Pixi) In P{ Xi ), (1) 

'— i, where Xi are the possible realizations of X. Relaxing the restrictions in the underlying Khinchin or Fadeev axioms it 
is possible to introduce the family of Renyi entropies 

N 

H M(X) = — ln^P^) 9 , (2) 

which in the case of Renyi order q = 2 are quite often used for estimation of entropies via the correlation sum Q . In 
the case of time series analysis it is a common task to evaluate the uncertainty of m random variables belonging to 
77i successive time steps Q • The corresponding joint entropy in the suitable notational representation reads 
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(3) 



In this expression e is the space resolution and r is the step length of the time discretization. After having decided 
for a suitable Renyi order, it is often omitted. Now the finite-m-entropy rate can be defined as 

, / \ #i|m(e,T) H m+1 (e,T) - H m (e,r) 

n m (e,T) := := . (4) 

T T 

This quantity is a special conditional entropy per time step length. Since obviously this entropy rate is obtained from 
a quotient of differences, a time-continuous formulation of the relationships of entropic quantities should naturally 
also be at hand. The idea of connection of the entropic quantities by derivatives is also supported in 5], where 
however, the time discretization step length r is still kept finite. The questions of consequences of variable time step 
length r and especially the limit of r — > for entropic quantities are addressed in this paper and it is intended to 
give theoretical insights into the structure of the relationships behind such entropic quantities. With those efforts 
this paper tries to contribute to a symmetrization of the treatment of entropies concerning space, where continuity is 
already realized in formulas, and time. 
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First, the line of thought from joint Shannon entropies to the Kolmogorov-Sinai entropy rate is outlined in sec. [TT1 
staying quite close to the paper of Gaspard and Wang |6| , the central paper behind this work. The limit of infinitesimal 
time step length r is performed after having performed the limit of infinite time t = mr. Theoretically, sec. lllll contains 
the central aim of this paper. It is the generalization of sec. HT1 with omission of the limit of infinite uncertainty-assessed 
time i, concentrating on the limit of infinitesimal time discretization step length t. In sec. HVI the presented ideas are 
tested numerically for the examples of the Roessler, Lorenz and Ornstein-Uhlcnbeck dynamics with the detection of 
qualitative discrepancies of deterministic and stochastic dynamics. Sec. fVl concludes the results of this paper. 

II. FROM SHANNON ENTROPY TO KOLMOGOROV-SINAI ENTROPY RATE 

The starting point as given in [6| is the joint Shannon entropy 

H(A,T,t) = H m — t / T (A,T) = - ^2 f(^o,-,w m _i)lnP(cJo,-,Wm-i) • (5) 

WOvA'm-l 

The symbol r, being the time step length, appears on the right side only implicitly as the time between successive 
realizations and the partition A appears on the right side implicitly in the range of values u>k, which corresponds to i 
in eq. ([T]), can take. In the following it is assumed that the partition for t\ is consistent with the partition for t 2 > t\. 
It is possible to define the rate 

h(A,r,t):=^f^. (6) 

The entropy per unit time with respect to partition A is then defined as the limit 

h(A,r) := lim h(A,T,t) . (7) 

t — >QO 

Starting from eq. (JSJ) it is also possible to define another rate 

ua A H(A,T,t + r)-H(A,T,t) 

h(A, r, t) := , (8) 



and the corresponding limit 



In general it holds 



h(A,r) := lim h(A,T,t) . (9) 

t — KX> 



h(A,T,t) ^ h(A,T,t) , but h(A, t) = h(A, t) . (10) 
The er-entropy rate is derived by 

h(e,r)= inf h{A,r) . (11) 

A.:diam(Ai)<e 

On the other hand, having obtained i/ m (e,r) from H m (A 7 r), e.g., again via infimum 

H m (e, T )= inf H m (A,r) , (12) 

A:aia7n[Ai )<e 

by the limit of infinite time from the conditional entropy rate also er-entropy rates can be obtained 0] : 

hoo(e,T)= lim h m (e,r) = lim -[H m+1 (e, t) - H m (e, t)] . (13) 

m — >oo m — >oo 7" 

Depending on the state space resolution e and time resolution t, ft.(e,r) and h ao (e,T) both are the uncertainty per 
time step of the immediate future time step ahead if infinite time conditioning is imposed. The Kolmogorov-Sinai 
entropy rate for processes in continuous time is obtained from 

h KS = lim h 00 {e,T)= lim h(e,r) . (14) 

e— >0,r— >0 e^0,r— >0 
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III. TIME-CONTINUOUS INFORMATION THEORY 



In the former section the succession of limit procedures for accessing the KS entropy rate was redisplayed. Starting 
with the same formula (0, it is a naturally arising question what happens if the limits of t — > oo and to — > oo 
are not performed, but instead the limit r — > is investigated 0. This does not lead to dynamical invariants, but 
instead to prediction-relevant quantities of information theory in the time-continuous case, because prediction deals 
with finite-time conditioning. 

The partition dependence of eq. (O or a similar resolution (e) dependence is suppressed in notation in this section, 
since the time aspects should be pointed out, and it remains the entropy H(r,t). Since i? m =i(r) is the uncertainty 
of the realization of one random variable, it should be independent of r. This results in 

H(t, t = t) = const for r > , (15) 

where a double r - dependence on the left side of the equation leads surprisingly to r-independence of the whole 
expression. 

For fixed t now the limit r — ► has to be carried out. If the limit exists, then 

H(t = 0,t) = lim H{r,t) (16) 

T— >0 

is the time-continuous joint entropy. Otherwise a time-continuous treatment of the joint uncertainty is not possible 
for the process at hand. Taking on the other hand the other involved variable t equal to zero it holds 

H(T,t = 0) = . (17) 

Hence it is inferred 



H{T = 0,t = 0)=0 . (18) 

From H m {r) = H(r,t = tot) being (not necessarily strong) monotonously increasing in to it is inferred that also in 
the limit of r — > the entropy H(t = 0,t) is (not necessarily strong) monotonously increasing in t. An interesting 
question is the finiteness of H(t = 0,t) for finite uncertainty-assessed time t for various process classes, which will 
be answered numerically by treating examples in sec. IIVI shown in fig. 's [21 HI [3 and [S] For fixed t > the performed 
limit causes 

lim to = lim — = oo . (19) 

From eqs. (O and (flT)]) it has to be inferred that H(t = 0, t) as the uncertainty of the whole path has finally to be 
understood in terms of path integral-type quantities, where the initial and final states are not fixed. 



Eq. ((3]) in notation suitable for the purpose of a time-continuous formulation or eq. J5]) without partition dependence 
read 



If the following limit exists, then 



MT,t) = g fr' + T >- g fr'>. (20) 



h(r = 0,t)= lim h(T,t) (21) 



is the finite time entropy rate in the time-continuous case. The discrepancy from the usual differentiation is that 
the function to be differentiated depends explicitly on the parameter of the differentiation. This is unusual, but not 
untreatable. In case of existence of H(t — 0, t) in eq. (fT6|) for the corresponding derivative it holds that 

h(T=o, t) = ^mt+**)-mt) (22) 

i.e., h(0,t) from eq. ((2"T|) is obtained as a usual derivative from H(0,t). h(r = At,t) is found to be a perturbation of 
the usual quotient of differences H (°' t + A ^~ H (°' t ) ; but the limit of A< — * is the same in both cases. From eqs. (|20[) . 
JT7J) and (dU) it is derived 



r • h(r, t = 0) = H (r, t = r) = const 



(23) 
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One concludes trivially that for arbitrary dynamics 

lim/i(r,t = 0) = 00 if H(t, t = r) > . (24) 



r — 



For deterministic dynamics the behaviour of the entropy rate h will be shown in fig. [5l It should be mentioned that 
the same treatment as here for the first derivative can of course also be carried through for higher derivatives Q ■ 
Eq. (|20|) inverted and iterated leads with 



H(t, t = mr) = {h{r, t=(m- l)r) + ... + h(r, 0)) • r (25) 
in the time-continuous limit to a representation of the joint entropy via integral: 

H(t = 0, [0,t]) = H( T = 0,t) 

771—1 

m — 1 
fc=0 

= lim / duh(T,u) 

/ du/i(r = 0,u) . (26) 
Jo 

Interchangeability of the limit procedures differentiation and integration is needed in the final step. 



IV. NUMERICAL CALCULATIONS FOR TIME-CONTINUOUS ENTROPIES 
A. Roessler system (deterministic chaotic dynamics) 

The Roessler system is given by 

x = -y - z , 
y = x + ay , 

z = b + z(x - c) . (27) 

The parameters are chosen as a = b = 0.2 and c = 5.7. The quadratic term 'zx' is the only nonlinearity. The largest 
Lyapunov exponent of the Roessler attractor is A ss 0.07 and the fractal dimension is = 1.99 ± 0.07. 

In fig. [T] the joint entropy of the x-coordinate of the Roessler system is given as a function of resolution e, time step 
length t and uncertainty-assessed time t. Convergence of the joint entropy for decreasing time step length t can be 
seen. The joint entropy H depends logarithmically on the resolution e, from which the dimension is obtained as a 
slope according to 

D M = _ Hm fl^S5i (28) 
m e^o me 

(H, p.106) with a value as predicted. 

It is possible to see in fig. [2] (in comparison with fig. Q] one smaller decade of resolutions is shown) that the z- 
coordinatc of the Roessler attractor carries a different uncertainty for the same values of (e, r, t) compared with the 
x-coordinate. Nevertheless the dimension of the attractor is captured also by the joint entropy of the z-coordinate. 

In fig. [3] the slice of fig. [T] for resolution e = 10 -2 with extended time t is shown. It is found that asymptotically in 
t the joint uncertainty of the Roessler system increases linearly. The measured slope for smallest r in between t = 20 
and t = 40 is 0.08. It estimates the KS entropy rate 

hxs = li m Ihn lirn h(e,T,t) . (29) 

e— >0 T— >0 t— »oo 
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FIG. 1: Joint entropies of the Roessler system as a function of the time t and the resolution e for various r = tq ■ 2 n ; n — 0, 



(to = 0.01) for the x- coordinate of the Roessler dynamics; 3 ■ 
approximately 3 circulations in the attractor. 



10 data points in the generated time series; t — 5 corresponds to 
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FIG. 2: Joint entropies of the Roessler system as a function of the time t and the resolution e for various r = To ■ 2 n ; n — 0, 8 
(to = 0.01) for the z-coordinate of the Roessler dynamics; 3 • 10 6 data points. 



The deviation from the expected value 0.07 can be found in the fact that still too small t or too large e are used for 
the estimation. It can be concluded from the small slope that the uncertainty of the first few points is larger than 
the uncertainty of a rather long motion in the attractor given the first few points. 

In fig. 2] the slice of fig. [2] for resolution e = 10~ 2 with extended time t is shown. Compared to fig. [3] a much 
smaller initial uncertainty Hi is observed for the z-coordinate. This is understood from the high probability of the 
z-coordinate to stay at zero. Furthermore in contrast to the x-coordinate a wave-like structure of the joint entropy for 
smaller time t is observed for the z-coordinate. A non-monotonous entropy rate ftasa function of t has to be inferred. 
This unintuitive signature was robustly reproduced under various parameter values and confidence in this result is 
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FIG. 3: Joint entropy of the x-coordinate of the Roessler system for fixed resolution e = 10 2 . With respect to time t it is an 
extended slice of fig. [1] for fixed resolution e. 




FIG. 4: Joint entropy of the z-coordinate of the Roessler system for fixed resolution e = 10 2 . With respect to time t it is an 
extended slice of fig. for fixed resolution e. 



caused by the fact that asymptotically for smallest available r the slope of 0.08 is found, which serves as a correct 
estimation of the KS entropy also from the z-coordinate of the Roessler system. Since also (from fig. [5J the dimension 
of the Roessler dynamics is correctly extractable the result should not be a numerical artefact. Nevertheless it is not 
expected that this should be true physics in the sense of increasing uncertainty by enlarged conditioning. According 
to Shannon entropies, for which additional conditioning cannot increase uncertainty, this behaviour is forbidden. A 
possible explanation of this behaviour can be assigned to effects of Renyi order q = 2 (cmp. eq. because in this 
case it seems unproven that increasing conditioning necessarily reduces the resulting entropy. With this reasoning 
the example carries the potential of tearing apart the interpretation of uncertainty from q ^ 1 - entropies. Another 
possible explanation that the wave-like structure corresponds to a finite sample effect being responsible for non- 
ergodicity has to be treated as rather improbable, because also for smallest resolution (largest e) it was not possible 
to detect a qualitative change in the sense of smoothing of the wave-like structure of the joint entropy under drastical 
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FIG. 5: Entropy rate as a function of resolution e, uncertainty-assessed time t and varying time discretization interval r for 
the x-coordinate of the Roessler system with parameters for chaotic dynamics. Upper panel: Focus on resolution dependence 
of divergent behaviour for small time t. Lower panel: Focus on larger times t for the Kolmogorov-Sinai entropy rate. 



enlargement of the length of the dataset. 

With consideration of the fact that the values of Hks and D were estimated for finite large t and finite small e 
instead of in the true limit it can be concluded that the same values are obtained from the x- and the z-coordinatc 
of the chaotic Roessler system in the limit cases, in principle in accordance with the theorem of Takens [lof . On the 
other hand, it can be clearly seen that for finite time t and finite resolution e the joint uncertainty is allowed to behave 
qualitatively different for different observables as the x- and the z-coordinate of the chaotic Roessler system. 

In fig. [5] the entropy rate of the Roessler system is shown. For every r the entropy rate as a function of t and e 
is represented by a surface. Those surfaces are interleaved for different r. Depending on the ranges shown along the 
axes different results become appercnt: With the example of the Roessler system even for deterministic dynamics a 
true divergence of h(r = 0,t = 0) can be observed for sufficiently small e seen in particular in the upper panel of 
fig. 03 The trivial argument of eq. already gave a hint for such behaviour. It is possible to see that h(e, r, t = 0) 
for finite r increases logarithmically in e with a slope depending on r. In the lower panel for larger t the entropy rate 
indicates a small finite value in the front right corner of the plot, which for sufficiently large t approximates the KS 
entropy rate. 
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B. Lorenz system (deterministic chaotic dynamics) 

The implemented discretized equations for the Lorenz dynamics are 

x n +i = x n + cr(-x n + y„)At , 

y n+1 = y n + {-x n z n + rx n - y n )At , 

z n +i = z n + (x n y n - bz n )At (30) 

with the usual parameters 

r = 28.0, a =10.0, &=f- (31) 
The fractal dimension of the Lorenz attractor is about 2.06 and the largest Lyapunov exponent is A « 0.91. 
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FIG. 6: Joint entropy of the Lorenz system as a function of the time t and the resolution e for various r = to ■ 2 n ; n = 0, 8 
(ro = 0.01) for the x-coordinate of the Lorenz dynamics; 3 • 10 6 data points. 

In fig. [5] the joint entropy of the Lorenz system is shown as a function of the resolution e, time step length r and 
uncertainty-assessed time t. As for the Roessler system it is possible to see the convergence of the joint entropy H 
for decreasing time step length r and a logarithmic dependence of the joint entropy on the resolution e with a slope 
giving the expected dimension in the limit of infinitesimal small e. 

In fig. [7] a slice of fig. [6] for fixed resolution is shown. An asymptotically linear behaviour in time can be found. 
The fact, that the slope is steeper than that of the chaotic Roessler attractor of fig. [3] is in accordance with the larger 
number of nonlinear terms and the known largest Lyapunov exponents. The slightly larger slope compared to the 
known KS entropy of the Lorenz system can be explained with estimation at rather large e. The appearance of finite 
sample fluctuations in entropy estimation in fig.[S]for small e, small r and large t under the same estimation conditions 
sooner in the Lorenz-case than in the Roessler-case is in accordance with the enhanced uncertainty H of the Lorenz 
dynamics. 

Furthermore fig. [7] indicates that the joint entropies for different finite r- values do not coincide for asymptotically 
large times t even though this cannot finally be proven in a plot for finite t. The slopes seem to reach the same value 
in all cases as already observed in fig. [3] for the x-coordinate of the Roessler dynamics, and hence the estimation of 
the KS entropy rate is rather independent of the choice of r, i.e., in the case of deterministic dynamics the limit with 
respect to r in eq. (fT4|) is not so important. On the other hand, the behaviour of H(t = 0,t) for finite rather small t 
is better resolved for smaller r, and this in general is rather important with respect to prediction. 
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FIG. 7: Joint entropy of the Lorenz dynamics for resolution e = 10 -2 (Slice of fig. EJ) ; 3 • 10 6 data points; to = 0.01. 

C. Ornstein-Uhlenbeck process (linear stochastic dynamics) 

The Ornstein-Uhlenbeck process 

X t = -aX t + y/DW t (32) 
can be numerically implemented by its discretisation 

X n+1 = (1 - aAt)X n + V~DVAi Vn+1 , Vn+1 ~ Af(0, 1) iid . (33) 

In fig. [8] the numerical analysis of the Ornstein-Uhlenbeck process indicates the non-existence of the limit 
lim T ^o H (e, r, t) for sufficiently small e and finite t except at t = for time- and amplitude-continuous stochas- 
tic dynamics with continuous trajectories. Whereas in the deterministic case, e.g. for the Lorenz dynamics, the value 
of the KS entropy rate was approximately seen as a slope at sufficiently small finite e also for finite r, in the time- 
and amplitude-continuous stochastic case the slope of H with respect to t asymptotically in t for sufficiently small 
finite e is seen in fig. [H] to be a function of r. Furthermore it was possible to show numerically that for sufficiently 
small e and non-zero finite fixed time t the entropy rate /i(e, T,t > 0) diverges logarithmically with decreasing t — > 
in the example of Ornstein-Uhlenbeck dynamics. Consistently, fig. [8] supports Iiks = oo for this stochastic dynamics, 
interestingly already from the r-behaviour. It should be mentioned here that for r > eq. (|20p can be seen as a 
naturally given regularization of the unavailable eq. (I22[) in the stochastic case. 

The author is aware of the fact that the result concerning the r-dependence contradicts (Q, p. 322) according to 
which processes with continuous realizations are said to have r-independent (e, r)-entropies /i(e, r) per unit time, 
which corresponds to the suggestion of finite lim r ^o H(e, t, t). The argument for this behaviour was essentially 
that for finite e a finite crossing time of underlying boxes of the partition leads to finite uncertainty for continuous 
trajectories (@, p. 317). On the other hand, an explanation for the numerically found behaviour in fig.[5]could be that 
infinite uncertainty arises from the trajectories close to the border of boxes of partitioning, where in the limit of t — ► 
infinitely often moving back and forth between boxes in finite time t is in principle possible for the Ornstein-Uhlenbeck 
process. This would create an infinite and hence dominant contribution for an averaged uncertainty calculation. 

From fig. [9] it is obtained that in the transition regime of large e the behaviour of non-existing lim r ^o H{e, r, t) is 
suppressed. The plots do not allow for the decision of the question if the transition from finite to infinite occurs at 
the resolution of the size of the system or at a smaller value for e. According to eq. (j2"5|) fig. [5] yields finite dimensions 
for finite r increasing in t for stochastic dynamics, even though the limit e — > can of course not be seen. In the limit 
of r — > 0, what is effectively like the limit of m — » oo, fig. O suggests an infinite dimension for stochastic dynamics. 

The results of this section are qualitatively robust against changes of the parameter a of the Ornstein-Uhlenbeck 
process and hence qualitatively robust against changes of the autocorrelation of the process. 
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FIG. 8: Joint entropy of the Ornstein-Uhlenbeck process. Parameters: a — 0.025; to = 10 4 ; 150000 data points. 




FIG. 9: The same plot as in fig. [8] from another point of view. 
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V. CONCLUSION 

The central objects of investigation in this paper are the entropy (e, r, t) and the entropy rate h^ q ' (e, t, t) as a 
function of the resolution e, the time discretization step length r and the uncertainty-assessed time t with numerical 
access to the case of Renyi order q = 2. Special focus is laid on the analysis of the behaviour of these quantities 
for varying r and especially on the time-continuous limit r — > while t is kept finite. In case of existence, in the 
time-continuous limit the entropy rates can be understood as usual time-derivatives of entropies. However, for finite 
r, in consequence of the explicit r-dependence of entropies, discrepancies from the usual quotient of differences are 
present. 

Numerically the analysis of time- and space-continuous dynamics was carried out with the Roessler and Lorenz 
system for the deterministic case and with the Ornstein-Uhlenbeck process for the stochastic case. Qualitative 
discrepancies of deterministic and stochastic dynamics were found. In the deterministic case the uncertainty H(e, t — 
0, t) is finite for all finite e and t. The entropy rate h(e, r = 0, t) becomes constant for large t and sufficiently small e. 
In the stochastic case the uncertainty H(e,r — 0, t) is infinite for all t > if e is below some threshold. Only for e 
above the threshold it is finite and then effectively like in the deterministic case. The stochastic result is qualitatively 
independent of the value of the width of the input noise as long as the width is not exactly zero, corresponding to 
determinism and introducing the qualitative change. This behaviour of entropies in the limit r — > seems to offer a 
new possibility for distinction of chaos and noise (see also 0]). 

The convergence of H in the limit r —> in the deterministic case can be interpreted as a saturation of the gain 
from higher sampling rates, such that a threshold sampling rate can be postulated, above which almost nothing more 
can be learned, i.e., in the deterministic case the continuous limit can be well approximated by discrete sampling with 
sufficiently high sampling rate. A criterion for an optimal sampling rate could be formulated. 

In the limit case e — > the partial derivative of the joint entropy H with respect to In e yields (minus) the dimension 
and if furthermore the limit t — > oo is carried out, the partial derivative of H with respect to t yields the KS entropy 
rate. All information concerning entropy rates (including the KS entropy rate) and dimensions of the dynamics can 
be extracted from one single plot of H^ q ' (e, r, i). In the deterministic case known values were verified in examples. 
For time- and amplitude-continuous stochastic dynamics it was seen that the partial derivative of H with respect to 
minus In e becomes infinite for t — > oo or r — ► and the partial derivative of H with respect to t becomes infinite for 
e -> or r -> 0. 

According to usual rules the total differential of the joint entropy H can be written as 

dH= — dt + ^r— dine. (34) 
at a m e 

For sufficiently small e and sufficiently large t this becomes 

dH = h KS dt- Ddlne . (35) 

It is possible to see that directional derivatives of H in general mix the properties of dimension and entropy rate. 
For integral quantities H(e,t) w h^st — D loge + const was suggested in ([11], p. 529), from which eq. (|33|) can be 
derived, but a prescription for the calculation of the constant is not given. Since eq. ([35]) avoids the offset problems 
it is a slightly reduced and hence preferable representation of the relationship of the involved quantities. 

Concerning the limit cases t — > oo and e — * with the determination of dynamical invariants, which is discussed in 
the literature, e.g. see (12], p. 1321), the time-continuous case treated in this paper gives a new unified representation 
of singly known results. On the other hand, at least of the same importance, the case of finite time t is of strong 
interest, if questions concerning optimal finite time conditioning with respect to prediction are addressed. A finite 
resolution e is needed for optimality of local prediction methods. With finite t and e, the limit r — > is of interest for 
the determination of the informational characteristic of the dynamics. 

A non-monotonous entropy rate for Renyi order q = 2, i.e., a conditional entropy, which does not monotonously 
decrease in the length of conditioning, is numerically found for the z-coordinate of the Roessler system in fig. [4] This 
could be interpreted as a hint for problems of the interpretation of uncertainty for entropies with q 1 in maximal 
generality. 
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